80 research outputs found
Teaching Compositionality to CNNs
Convolutional neural networks (CNNs) have shown great success in computer
vision, approaching human-level performance when trained for specific tasks via
application-specific loss functions. In this paper, we propose a method for
augmenting and training CNNs so that their learned features are compositional.
It encourages networks to form representations that disentangle objects from
their surroundings and from each other, thereby promoting better
generalization. Our method is agnostic to the specific details of the
underlying CNN to which it is applied and can in principle be used with any
CNN. As we show in our experiments, the learned representations lead to feature
activations that are more localized and improve performance over
non-compositional baselines in object recognition tasks.Comment: Preprint appearing in CVPR 201
Fast exploration and learning of latent graphs with aliased observations
We consider the problem of recovering a latent graph where the observations
at each node are \emph{aliased}, and transitions are stochastic. Observations
are gathered by an agent traversing the graph. Aliasing means that multiple
nodes emit the same observation, so the agent can not know in which node it is
located. The agent needs to uncover the hidden topology as accurately as
possible and in as few steps as possible. This is equivalent to efficient
recovery of the transition probabilities of a partially observable Markov
decision process (POMDP) in which the observation probabilities are known. An
algorithm for efficiently exploring (and ultimately recovering) the latent
graph is provided. Our approach is exponentially faster than naive exploration
in a variety of challenging topologies with aliased observations while
remaining competitive with existing baselines in the unaliased regime
Query Training: Learning a Worse Model to Infer Better Marginals in Undirected Graphical Models with Hidden Variables
Probabilistic graphical models (PGMs) provide a compact representation of
knowledge that can be queried in a flexible way: after learning the parameters
of a graphical model once, new probabilistic queries can be answered at test
time without retraining. However, when using undirected PGMS with hidden
variables, two sources of error typically compound in all but the simplest
models (a) learning error (both computing the partition function and
integrating out the hidden variables is intractable); and (b) prediction error
(exact inference is also intractable). Here we introduce query training (QT), a
mechanism to learn a PGM that is optimized for the approximate inference
algorithm that will be paired with it. The resulting PGM is a worse model of
the data (as measured by the likelihood), but it is tuned to produce better
marginals for a given inference algorithm. Unlike prior works, our approach
preserves the querying flexibility of the original PGM: at test time, we can
estimate the marginal of any variable given any partial evidence. We
demonstrate experimentally that QT can be used to learn a challenging
8-connected grid Markov random field with hidden variables and that it
consistently outperforms the state-of-the-art AdVIL when tested on three
undirected models across multiple datasets
Schema-learning and rebinding as mechanisms of in-context learning and emergence
In-context learning (ICL) is one of the most powerful and most unexpected
capabilities to emerge in recent transformer-based large language models
(LLMs). Yet the mechanisms that underlie it are poorly understood. In this
paper, we demonstrate that comparable ICL capabilities can be acquired by an
alternative sequence prediction learning method using clone-structured causal
graphs (CSCGs). Moreover, a key property of CSCGs is that, unlike
transformer-based LLMs, they are {\em interpretable}, which considerably
simplifies the task of explaining how ICL works. Specifically, we show that it
uses a combination of (a) learning template (schema) circuits for pattern
completion, (b) retrieving relevant templates in a context-sensitive manner,
and (c) rebinding of novel tokens to appropriate slots in the templates. We go
on to marshall evidence for the hypothesis that similar mechanisms underlie ICL
in LLMs. For example, we find that, with CSCGs as with LLMs, different
capabilities emerge at different levels of overparameterization, suggesting
that overparameterization helps in learning more complex template (schema)
circuits. By showing how ICL can be achieved with small models and datasets, we
open up a path to novel architectures, and take a vital step towards a more
general understanding of the mechanics behind this important capability
- …